Information-Theoretic Approach to Interpret Internal Representations of Self-Organizing Maps

In this chapter, we propose a new method to measure the importance of input variables and to examine the effect of the input variables on other components. We applied the method to competitive learning, in particular, self-organizing maps, to demonstrate the performance of our method. Because our method is based upon our information-theoretic competitive learning, it is easy to incorporate the idea of the importance of input variables into the method. In addition, by using the SOM, we demonstrate visually how the importance of input variables affects the outputs from the other components, such as competitive units. In this section, we first state that our objective is to interpret the network configurations as clearly as possible. Then, we show why the importance of input variables should be taken into account. Finally, we will briefly survey our information-theoretic competitive learning and its relation to the importance of input variables. The objective of the new method is to interpret network configurations, focusing upon the meaning of input variables in particular, because we think that one of the most important tasks in neural learning is that of interpreting network configurations explicitly (Rumelhart et al., 1986; Gorman & Sejnowski, 1988). In neural networks’ applications, we have had much difficulty to explain how neural networks respond to input patterns and produce their outputs due to the complexity and non-linear nature of data transformation (Mak & Munakata, 2002), namely, the low degree of human comprehensibility (Thrun, 1995; Kahramanli & Allahverdi, 2009) in neural networks. One of the major approaches for interpretation is rule extraction from trained neural networks by symbolic interpretations with three types of methods, namely, decompositional, pedagogical and eclectic (Kahramanli & Allahverdi, 2009). In the decompositional approach (Towell & Shavlik, 1993; Andrews et al., 1993; Tsukimoto, 2000; Garcez et al., 2001), we analyze the hidden unit activations and connection weights for better understanding of network configurations. On the other hand, in the pedagogical approach (Andrews et al., 1993), the neural network is considered to be a black box, and we only focus upon the imitation of input-output relations exhibited by the neural networks. Finally, in the eclectic approach (Andrews et al., 1993; Barakat & Diederich, 2005), both pedagogical and decompositional approaches are incorporated. In the popular decompositional approach, much attention has been paid to hidden units as well as connection weights. The importance of input variables has been implicitly taken into account. For example, Tsukimoto (Tsukimoto, 2000) used the absolute values of connection weights or the squared connection weights to input variables (attributes) for measuring the importance of input variables. In addition, 1